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OPERATING SYSTEMS 

This invention relates to operating systems. More particiilarly, this invention 
relates to systems, methods and computer programs for running multiple operating 
systems concurrentiy, 

5 For some computer programs, it is critical tihiat steps in the program are 

performed within defined time periods, or at defined times. Examples of such 
programs are control programs for operating mobile telephones, or for operating 
private branch exchanges (PBXs) or cellular base stations. Typically, the program 
must respond to external events or changes of state in a consistmt way, at or within a 

10 certain time after the event. This is referred to as operating in "real time". 

For many other programs, however, the time taken to execute the program is 
not critical. This applies to most common computer programs, including spreadsheet 
program, word processing programs, payroll packages, and general reporting or 
analysis programs. Qa the other hand, whilst the exact time taken by such programs is 

15 not critical, in most cases, users would prefer quicker execution where this is possible. 

Applications programs interact with the computers on which they nm through 
operating systems. By using the applications programming interface (API) of the 
operating system, the applications program can be written in a portable fashion, so that 
it can execute on different computers with different hardware resources. Additionally, 

20 conmion operating systems such as Linux or Windows provide multi-tasking; in other 
words, they allow several program to operate concuirenfly. To do so, they provide 
scheduling; in other words, they share the usage of the resources of the computer 
between the different programs, allocating time to each in accordance with a, 
scheduling algorithm. Operating systems of the this kind are very widely used, but 

25 they generally make no provision for running real time applications, and they therefore 
are unsuitable for many control or communications tasks. 

For such tasks, therefore, real time operating systems have been developed; one 
example is ChorusOS Calso know as Chorus) and its derivatives. Chorus is available 
as open source software Jfrom: 

30 http://www.e3q)erimentalstufif.com/Technologies/ChonisOS/ind 
and Jaluna at 
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htfp://www.jaluna,coin/ 

It is described in "ChorusOS Features and Architecture overview" Francois 
Armand, Sxm Technical Report, August 2001, 222p, available from: 
http://www.jaluna,cora/developer/papers/COSDESPERF.pdf 
S These operating systems could also be used to run other types of programs. 

However, users understandably wish to be able to run tihie vast number of "legacy" 
programs which are written for general purpose operating systems such as Windows ox 
Linux, without having to rewrite them to run on a real time operating system. 

It would be possible to provide a "dual boot" system, allowing the user to run 

10 either one operating system or the other, but there are many cases where it would be 
desirable to be able to run a "legacy" program at the same time as running a real time 
program. For example, telecommunications network infrastructure equipment, third 
generation mobile phones and other advanced phones, and advanced electronic gaming 
equipment may require both realtime applications (e.g. game playing graphics) and 

15 noh-realtime applications (game download). 

Li US 5903752 and US 5721922, an attempt is made to incorporate a real time 
environment into a non real tune operating system by providing a real time multi- 
tasking kernel in the intemq)t handling environment of the non real time operating; 
system (such as Windows). 

20 One approach which has been widely used is "emulation". Typically, an. 

emulator program is written, to run under the real time operating system, which, 
interprets each instmction of a program written for a general purpose operating system^, 
and performs a corresponding series of instructions xmder the real time operating 
system. However, since one instruction is always replaced by many, emulation places 

25 a heavier load on the computer, and results in slower performance. Similar problems 
arise from the approach based on providing a virtual machine (e.g. a Java™ virtual 
machine). Examples of virtual machine implementations are EP 1059582, US 
5499379, and US 4764864. 

A further similar technique is described in US 5995745 (Yodaiken). Yodaikea 

30 describes a system in which a multi tasking real time operating system runs a general 
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purpose operating system as one of its tasks, pre-empting it as necessary to perform 
real time tasks. 

Another approach is to run the realtime operating S3^tem as a module of tihe 
general purpose operating system, as described in for example EP 0360135 and the 
5 article ^Merging real-time processing and UNIX V", (Gosch), ELECTROKICS, 
September 1990 p62, Tn this case, hardware intemipts are selectively masked with the 
intention that those concerned with the general purpose operating system should not 
pre-empt the realtime operating system. 

Another approach is that of ADEOS (Adaptive Domain Environment for 
10 Operating Systems), described in a White Paper at 
http://opersysxona/ftp/pub/Adeos/adeos.pdf 

ADEOS provides a nanokemel which is intmded, amongst other things, for 
running multiple operating systems although it appears only to have been implemented 
with Linux. One proposed use of ADEOS was to allow ADEOS to distribute 
15 interrupts to RTAI (Realtime Application Interface for Linux) for which see: 
uttp;/Vwww.aero.poiinu.iT/~rtai/^pucations/. 

EP 1054332 describes a system in which a "switching unit" (which is not 
described in sufficient detail for full understanding) runs a realtime and a general 
purpose operating system. Hardware interrupts are handled by a common int^rupt 

20 handler, and in some embodiments, they are handled by the realtime op^ting system, 
which then generates software interrupts at a lower priority level which are handled by 
routines in the secondary operating system. 

An object of the present invention is to provide an improved system, m-Cthod 
and computer program for running multiple operating systems simultaneously, even 

25 when the systems are desigaed for different purposes. In particular, the present 
invention aims to allow one of the operating sj^ems (for example, a real time 
operating systems) to perform without disturbance, and the other (for example, a 
general purpose operating system) to perform as well as possible using the remaining 
resources of the computer. 

30 Accordingly, in one aspect, the present invention provides a system in which 

multiple operating systems are slightly modified and provided with a common program 



wo 2005/033928 



PCT/IB2004/003344 



4 

which schedules between them, in which one of the operating systems (the "primary" 
or "critical" operating system) is favoured over another (the "secondary" or non- 
critical operating system). Preferably, the invention allocates hardware preferentially 
to the critical operating system, and it denies the secondary operating systenx or 
5 systems access which would interfere with that of the critical operating systCTi. 
Preferably, the present invention uses the critical operating system drivers to access 
shared resources, even if the access is requested by the secondary operating system. 
However, in no sense is the critical operating system "nmning" the secondary 
operating system, as in US 5995745; each system ignores the others nmning alongside 

10 it and only communicates with the common program (corresponding to a nanokemel 
of the prior art) which brokers the access to the drivers of the critical operating system. 

Preferably, the secondary operating systems are modified so that they cannot 
mask intermpts, and their interrupt service routines are modified to make ttiem 
responsive to messages indicating that an interrupt occurred. The common program 

15 handles all hardware exceptions by passing them to the interrupt service routines of the 
primary operating system, and where a hardware intermpt was intended for one of the 
secondary operating systems, an interrupt message or notification is generated. NText 
time that secondary operating system is scheduled by the common program, the 
message or notification is passed to it, and the common program calls its interrupt 

20 service routine to service the intermpt. 

Thus, the secondary operating systems caimot pre-empt the primary operating 
system (or, in general, a higher importance secondary operating system) in any way on 
occurrence of an intermpt, since all are initially handled by the primary operating 
system and only notified to the secondary operating system for which they are destined 

25 after the primary operating system has finished execution and that secondary operating 
system is scheduled. 

Handling of such intermpts is thus deferred imtil no critical task in the prinxary 
op^ting system is occurring. When fliey are eventually actioned, however, the 
routines of the secondary operating system may operate substantially unmodified 

30 £ishion so that the behaviour is (except for the delay) as e?q)ected by the secondary 
operating s>^tem. 
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Other aspects, embodiments and preferred features, with corresponding 
advantages, will be apparent from the following description, claims and drawings. 

Embodiments of the invention will now be described, by way of example only, 
with reference to the accompanying drawings, in which: 
5 Figure 1 is a block diagram showing the clients of a computer system on 

which the present invention can execute; 

Figure 2a is a diagram illustrating the arrangCTient of software in the prior art; 

and 

Figure 2b is the corresponding diagram illustrating the arrangement of software 
1 0 according to the present embodiment; 

Figure 3 is a flow diagram showing the stages in creating the software of 
Figure 2b for the computer of Figure 1 ; 

Figure 4 show the components of a hardware resovirce dispatcher forming part 
of Figure 2b; 

15 Figure 5 illustrates the program used in a boot and initialisation sequence; 

Figure 5 illustrates the system memory image used in the boot or xmLialisatioii 
process; 

Figure 7 illustrates the transition from a primary operating system to a 
secondary operating system; 
20 Figure 8 illustrates the transition from a secondary operating system to a 

primary operating system; 

Figure 9a illustrates the communication between applications ruiming on 
different operating systems according to the invention; 

Figure 9b illustrates the communication between applications running on 
25 different operating systems on different computers according to the invention; 

Figure 10 shows an example of the primary, secondary and nanokemel virtual 
address spaces; 

Figure 1 1 shows how the memory context is switching in time; 
Figure 12 illustrates the visible part of the nanokemel context; and 
30 Figure 13 shows the execution flow and how the nanokemel stack is used to 

allow intermpt handling and primary kernel re-entrance. 
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Introduction 
System Hardware 

S A computer system to which the system is applicable 100 comprises a central 

processing xmit (CPU) 102, such as a Pentium 4™ CPU available from Intel 
Corporation, or PowerPC CPU available from Motorola (the embodiment has been 
implemented on both), coiqpled via a system bus 104 (comprising control, data and 
address buses) to a read-only memory (ROM) chip 106; one or more banks of random 

10 access memory (RAM) chips (108); disk controller devices 110 (for example IDE or 
SCSI controllers, connected to a floppy disk drive, a hard disk drive, and additional 
removable media drives such as DVD drives); one or more input/output ports (112) 
(for example, one or more USB port controllers, and/or parallel port controllers for 
coimection to printer and so on); an expansion bus 114 for bus connection to extemal 

15 or intemal peripheral devices (for example the PCI bus); and other system chips 116 
(for example, graphics and sound devices). Examples of computers ox this type are 
personal computers (PCs) and workstations. However, the application of the invention 
to other computing devices such as mainframes, embedded microcomputers in control 
systems, and PDAs (in which case some of the indicated devices such as disk drive 

20 controllers may be absent) is also disclosed herein. 

Management of Software 

Referring to Figure 2a, in use, the compute 100 of Figure 1 runs resident 
programs comprising operating system kemel 202 (which provides the output routines 

25 allowing access by the CPU to the other devices shown in Figure 1); an operating 
system user interface or presentation layer 204 (such as X Windows); a middleware 
layer 206 (providing networking software and protocols such as, for instance, a TCP/IP 
stack) and applications 208a, 208b, which run by making calls to the API routines 
forming the operating system kemel 202. 

30 The operating system kemel has a number of tasks, in particular: 
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■ scheduling (i.e., sharing the CPU and associated resources between different 
applications which are running); 

■ memory management (i.e. allocating mraiory to each task, and, where 
necessary, swapping data and programs out of memory add on to disk: drives); 

■ providing a file system; 

■ providing access to devices (typically, throu^ drivers); 

■ intenrupt handling; 

■ providing an applications programming interface enabling the 2q)plications to 
interact Avith system resources and users. 

The kernel may be a so-called "monolithic kernel" as for Unix, in w^hich case 
the device drivers form part of the kernel itself. Alternatively, it may be a 
"microkernel" as for Chorus, in which case the device drivers are separate of the 
kernel. 

In use, then, when the computer 100 is started, a bootstrap program stored in 
ROM 106 accesses the disk controllers 110 to read the file handling part of the 
operating system from permanent storage on disk into RAM 108, then loads the 
remainder of the operating system into an area of RAM 108. The operating system 
then reads any applications firom the disk drives via the disk controllers 110, allocates 
space in RAM 108 for each, and stores each application in its allocated memoxy space. 

During operation of the applications, the scheduler part of the operating system 
divides the use of tiie CPU between the different applications, allowing each a share of 
the tune on the processor according to a scheduling poUcy. It also manages nse of the 
memory resources, by "swapping out" infi-equently used applications or data (i.e. 
removing them firom RAM 108 to firee up space, and storing them on disk). 

Finally the routines making up the applications programming interfa-ce (API) 
are called from the appUcations, to execute fimctions such as input and output:, and the 
interrupt handling routines of flie operating system respond to interrupt and events. 

Summary of Principles of the Preferred Enibodiment 

In tiiie preferred embodiment, each operating system 201, 202 to be used on the 
computer 100 is sli^tly re-written, and a new low-level program 400 (termed here flie 
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"hardware resource dispatcher", and sometimes known as a "nanokemel" although it is 
not the kernel of an operating system) is created. The hardware resource dispatcher 
400 is specific to the particular type of CPU 102, since it interacts with the processor. 
The versions of the operating systems which are modified 201, 202 are also those 
5 which are specific to the hardware, for reasons which will become apparent. 

The hardware resoiirce dispatcher 400 is not itself an operating system. It does 
not interact with the appUcations programs at all, and has very limited functionality. 
Nor is it a virtual machine or emulator; it requires the operating sj^tems to be modified 
in order to cooperate, even though it leaves most of ttie processing to the operating 
10 systems themselves, nmning their native code on the processor. 
It performs the following basic functions: 

■ loading and starting each of the mxiltiple operating systems; 

■ allocating memory and other system resources to each of the operating systems; 

■ scheduling the operation of the different operating systems (i.e. dividing CPU 
15 time between them, and managing the change over between them); 

■ providing a **virtualised device" method of indirect access to those syst^n 
devices which need to be shared by the operating systems ('Virtualising" the 
devices); 

■ providing a communications link between the operating systems, to allow 
20 applications running on different opa:ating sj^tems to commimicate with each 

other. 

The operating systems are not treated equally by the embodiment. Instead, one 
of the operating systems is selected as the "critical" operating systems (this will be the 
real time operating system), and the or each other operating system is treated as a "non 
25 critical" or "secondary" operating systems (this will be the or each general purpose 
operating system such as Linux). 

When the hardware resoxurce dispatcher is designed, it is provided with a data 
structure (e.g. a table) listing the available system resources (i.e. devices and memory), 
to enable as many system devices as possible to be statically allocated exclusively to 
30 one or other of the operating systems. 
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For example, a parallel printer port might be statically allocated to the general 
purpose operating system 202, which will often run applications which will need to 
produce printer output. On the other hand, an ISDN digital line adapter port may be 
permanCTitly allocated to the real time operating system 201 for communications. This 

S static allocation of devices wherever possible means that each operating system can 
use its existing drivers to access statically allocated devices without needing to call the 
hardware resource dispatcher. Thus, there is no loss in execution speed in accessing 
such devices (as there would be if it acted as a virtual machine or CTiulator). 

In the case of system devices which must be shared, the hardware resource 

10 dispatcher virtuaUses uses of the devices by the non-critical operating systems, and 
makes use of the drivers supplied with the critical operating system to perform the 
access. Likewise, for interrupt handling, the interrupts pass to the critical operating 
system interrupt handling routines, which either deal with the interrupt (if it was 
intended for the critical operating system) or pass it back through the hardware 

15 resource dispatcher for forwarding to a non critical operating system (if that was where 
it was destined). 

On boot, the hardware resource dispatcher is first loaded, and it then loads each 
of the operating systems in a predetermined sequence, starting with the critical 
operating system, then following with the or each secondary operating system in turn. 
20 The critical operating system is allocated the resources it requires firom the table, and 
has a fixed memory space to operate in. Then each secondary operating system in turn 
is allocated the resources and m^ory space it requires fiiom the available remaining 
resources. 

Thus, according to the embodiment, the resources used by the operating 
25 systems are separated as much as physically possible, by allocating each its own 
memory space, and by providing a static allocation of devices exclusively to the 
operating systems; only devices for which sharing is essential are shared. 

In operation, the hardware resource dispatcher scheduler allows the critical 
operating system to operate until it has concluded its tasks, and then passes control 
30 back to each non critical operating system in turn, until the next interrupt ox* event 
occurs. 
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The embodiment thus allows a multi operating system environment in which 
the operation of the critical operating system is virtually unchanged (since it ixses its 
original drivers, and has first access to any interrupt and event handling). The 
secondary operating systems are able to operate efSciently, within the remaining 
processor time, since in most cases they will be using their own native drivers, and will 
have exclusive access to many of the system devices. Finally, the hardware resource 
dispatcher itself can be a small program, since it handles only hmited functions, so that 
system resources are conserved. 

The preferred embodiment is also economic to create and maintain, because it 
involves only limited changes to standard commercial operating systems which will 
already have been adapted to the particular computer 100. Fxu1;her, since the chianges 
to the operating systems are confined to architecture specific files handling matters 
such as interrupt handling, and configuration at initialising time, which interface with 
the particular type of computer 100, and which are unlikely to change as frequently as 
the rest of the operating system, there may be little or no work to do in adaptiag new 
versiuns of the same operating system to work in a multiple operating system fasliion. 

DetaOed Description of the Preferred Embodiment 

In this embodiment, the computer 100 was an Intel 386 family processor (e.g. a 
20 Pentium processor) and a Motorola PowerPC 750 (Reduced Instruction Set Computer 
or "RISC") computer (step 302). The critical operating system 201 was the C5 
operating system (the real time microkCTiel of Jalima-1, an open-source version of the 
fifth generation of the ChomsOS system, available for open source, free download 
from http://www.jaluna.com). 
25 In step 306, the ChorusOS operating system k^nel 201 is modified for 

operating in multiple operating system mode, which is treated in the same way s 
porting to a new platform (i.e. writing a new Board Support Package to allow 
execution on a new computer with the same CPU but different system devices). The 
booting and initialisation sequences are modified to allow the real time operating 
30 system to be started by the hardware resource dispatcher, in its allocated memory 
space, rather than starting itself. The hardware-probing stage of the initialisation 
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sequence is modified, to prevent the critical operating system from accessing the 
hardware resources which are assigned to other secondary sj^tems. It reads the static 
hardware allocation table from the hardware resource dispatcher to detect the devices 
available to it. 

Tnqp calls 2012 are added to the critical operating system, to detect states and 
request some actions in response. A trap call here means a call which causes the 
processor to save the current context (e.g. state of registers) and load a new context 
Thus, where virtual memory addressing is used, the address pointers are changed. 
For example, when the real time operating system 201 reaches an end point (and 
ceases to require processor resources) control can be passed back to the hardware 
resource dispatcher, issuing the "idle"' trap call, to start the secondary operating 

system. Many processors have a "halt" instruction. In some cases, only 
supervisor-level code (e.g. operatiug systems, not appUcations) can include such a 
'lialt" instruction, hi this embodiment, all the operating systems are rewritten to 
remove •*halt" iostructions and replace them with an "idle" routine (e.g. an execution 
ihxeau) which, when called, issues the ''idle'* tr^ call. 

Some drivers of the Board Support Package are specially adapted to assist the 
hardware resource dispatcher in virtualizing the shared devices for secondary operating 
systems. 

Additional *^drtual'* drivers 2014 are added which, to tiie operating system, 
appear to provide access to an input/output (I/O) bus, allowing data to be written to the 
bus. In fact, the virtual bus driver 2014 uses memory as a communications medium; it 
exports some private memory (for input data) and imports memory exported by other 
systems (for output data). In this way, the operating system 201 (or an application 
running on the operating system) can pass data to another operating system (or 
application running on it) as if they were two operating systems running on separate 
machines comiected by a real I/O bus. 

The secondary opiating system 202 was selected (step 308) as Linux, having a 
kernel version 2.4.18 (step 308). 

In step 310, the secondary operating system kernel 202 is modified to allow it 
to fimction in a multiple operating system environment, which is treated as a new 
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hardware architecture. As in step 306, the boot and initialisation sequences are 
modified, to allow the secondary operating system to be started by the hardware 
resource dispatcher, and to prevent it firom accessing the hardware resources assigned 
to the other systems, as specified in the hardware resource dispatcher table. As in step 
5 306, trap calls 2022 are added, to pass control to the hardware resource dispatcker. 

Native drivers for shared system devices are replaced by new drivers 2028 
dealing with devices which have been virtualized by the hardware resource dispatcher 
(intermpt controller, I/O bus bridges, the system timer and the real time clock). These 
drivers execute a call to virtual device handlers 416 of the hardware resource 

10 dispatcher in order to perform some operations on a respective device of the computer 
100. Each such virtual device handler 416 of the hardware resource dispatcher is 
paired with a "peer" driver routine in the critical operating system, which is arranged 
to directly interact with the system device. Thus, a call to a virtual device handler is 
relayed up to a peer driver in the critical system for that virtualized device, in order to 

15 make real device access. As in step 306, read and write drivers 2024 for the virtual 
L^O bus are provided, to allow iuter-operating system communications. 

The interrupt service routines of the secondary operating system are modified, 
to provide virtual interrupt service routines 2026 each of which responds to a 
respective virtual interrupt (in the form of a call issued by an interrupt handler routine 

20 412 of the hardware resource dispatcher), and not to respond to real interrupts or 
events. Routines of the secondary operating system (including intenvpt service 
routines) are also modified to remove masking of hardware interrapts (at least in all 
except critical operations). In that way, the secondary operating systems 202, ... are 
therefore pre-emptable by the critical operating system 201; in other words, the 

25 secondary operating system response to a virtual interrupt can itself be intermpted by a 
real interrupt for the critical operating system 201 . This typically includes: 

■ masking/unmasking events (interrapts at processor level); 

■ saving/restoring ev^ts mask status; 

■ identifying the intermpt source (interrupt controller devices); 

30 ■ masking/unmasking interrupts at source level (intermpt controller devices). 
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New virtual device drivers 2028 are added, for accessing the shared hardware 
devices (the I/O bus bridges, the system console, the system timer and the real time 
clock). These drivers execute a call to virtual device handlers 416 of the hardware 
resource dispatcher in order to write data to, or read data fronti, a respective device of 
5 the computer 1 00. 

To effect this, the Linux kernel 207 is modified in this embodiment by adding 
new virtual hardware resource dispatcher architecture sub trees (nk-i386 and nk-ppc 
for the 1-386 and PowerPC variants) wifli a small number of modified files. 
Unchanged files are reused in their existing form. The original sub-trees are retained, 
10 but not used. 

In step 312, the hardware resoxirce dispatcher 400 is written. The hardware 
resource dispatcher comprises code which provides routines for the following 
functions as (as shown in Figure 4): 

■ booting and initialising itself (402); 

15 ■ storing a table (403) which stores a list of hardware resources (devices such as 

ports) and an aliocation entry indicating to which operating system each 
resource is uniquely assigned; 

■ booting and initialising the critical operating system that completes the 
hardware resource dispatcher allocation tables (404); 

20 ■ booting and initialising secondary operating systems (406) 

■ switchiug between operating systems (408); 

■ scheduling between operating systems (4 1 0); 

■ handling interrupts (using the real time operating system interrupt service 
routines, and supplying data where necessary to the virtual interrupt service 

25 routines of the secondary operating systems) (412); 

■ handling trap calls firom each of the operating systems (414); 

■ handling access to shared devices from the secondary operating systems (416); 

■ handling inter-op^ating system communications on the virtual I/O bus (41 8). 
In fiulher embodiments (described below), it may also provide a system debugging 

30 framework. 



wo 2005/033928 



PCT/IB2004/003344 



14 

Operating system switcher 408 

In order to switch from an operatmg syst^ to another, the operating system 
switch^ 408 is arranged to save the "contexf' - the current values of the set of state 
variables, such as register values - of the currently executing operating system; xestore 
the stored context of another operating system; and call that other operating system to 
recommence execution where it left off. Where the processor uses segments of 
memory, and virtual or indirect addressing techniques, the registers or data struictures 
storing the pointers to the current memory spaces are thus swapped. For example, the 
operating systems each operate in dijBFerent such memory spaces, defined by the 
context including the pointer values to those spaces. 
In detail, the switcher provides: 

• explicit switches (e,g. trap calls) from the currently nmning to the next scheduled 
operating systems, when the current becomes idle; and 

• implicit switches from a secondary operating system to the critical operating 
system, when a hardware interrupt occurs. 

The switches may occur on a trap call or a real or virtual interrupt, as desoribed 
below. 

Scheduler 410 

20 The scheduler 410 allocates each operating system some of the available 

processing time, by selecting which secondary operating system (if more than one is 
present) will be switched to next, after exiting another operating system. In this 
embodiment, each is selected based on fixed priority scheduling. Other embodiments 
allowing specification based on time sharing, or guaranteed minimum percentage of 

25 processor time, are also contemplated herein. In each case, however, the critical 
operating system is pre-empted only when in the idle state. 

In finrther embodiments, the critical operating system may explicitly inform the 
scheduler 410 when it may be pre-CTq[>ted, so as to allow all secondary opeirating 
systems some access to the CPU to perform tasks with higfher priority then the tasks 

30 still r unning in critical system. Thus, in one example, the intemq>t service routines of 
the critical operating system cannot be pre-empted, so that the critical operating system 
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can always respond to external events or timing signals from the realtime clock, 
maintaining realtime operation. 

Handling virtualised processor exceptions 

The hardware resource dispatcher is arranged to provide mechanisms to handle 
processor exceptions (e.g. CPU interrupts or co-processor interrupts) as follows: 

• firstly, to intercept processor exceptions through the critical operating system; 

• secondly, to post a corresponding virtual exception to one or more secondary 
operating systems; to store that data and, when the scheduler next calls that 
secondary operating system, to call the corresponding virtual interrupt service 
routine 2026 in the secondary operating system; 

• thirdly, to mask or unmask any pending virtual exertions from within 
secondary operating systems. 

Virtualised exceptions are typically used for two different purposes; 

• Firstly, to forward hardware device interrupts (which are delivered as 
asynchronous processor exceptions) to secondary operating systems; 

• Secondly, to implement inter-operating system cross-intermpts - i.e. intermpts 
generated by one system for another intemxpts (which are delivered as 
synchronous exceptions). 

Trap call handler 414 

The operation of the trap call handler will become apparent from the following 
description. Its primary purpose is to allow the scheduler and switcher to change to 
another operating system when a first one halts (and hence does not require CPU 
resources). An additional role is to invoke hardware resource dispatcher services such 
as a system console for use in debugging as discussed in relation to later embodiments. 

Virtualised devices 416 

As indicated above, for each shared device (e.g. interrupt controller, bus 
bridges, system timer, realtime clock) each operating system provides a device driver, 
forming a set of peer-level drivers for tiiat device. The realtime operating system 
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provides the driver used to actually access the device, and the others provide virtual 
device drivers. 

The shared device handler 416 of the hardware resource dispatcher provides a 
stored data structure for each device, for access by all peer device drivers of tiiat 
5 device. When tiie device is to be accessed, or has been accessed, the device drivers 
update the data stored in the corresponding data structure with the details of the access. 
The peer drivers use cross-interrupts (as discussed above) to signal an event to notify 
other peer drivers that that the data structure has just been updated. 

The drivers which are for accessing interrupt controller devices use the 
10 virtualised exception mechanisms discussed above to handle hardware interrupts as 
follows: 

• The critical operating system device driver handles hardware interrupts and 
forwards them as virtualised exertions to the secondary peer drivers; 

• The secondary operating system enables and disables interrupts by using the 
1 5 virtualised exception masking and uimiasldng routines discussed above. 

I/O buses and their bridges only have to be shared if the devices connected to 
them are not all allocated to the same operating system. Thus, in allocating devices, to 
the ext^t possible, devices connected to the same I/O bus are allocated to the same 
operating system. Wh^e sharing is necessary, the resource allocation table 404 stores 
20 descriptor data indicating the allocation of the resources on the bus (address spaces, 
interrupt lines and I/O ports) to indicate which operating system has which resources. 

Implementation of the embodiment 

Finally, in step 314, the code for the hardware resource dispatcher and 
25 operatiQg systems is compiled as a distributable binary computer program product for 
supply with the computer 100. 

A product which may be siq)plied in accordance with an aspect of the invention 
is a developm^t enviromnent product, comprising a computer program which enables 
the user to select different operating systems to be used, build and select different 
30 applications for each operating system, embed the application and operating systems 
into a deliverable product, and provide for booting of the operating system and launch 
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of executable binaries of the applications. This is based on, and shnilar to, the C5 
development environment, available from wwwjaluna.com. 

Operation of the Embodiment During Booting and Initialisation 

5 Referring to Figure 5, the boot and initialisation processes according to thus 

embodiment are performed as follows. 

A bootstr^ping program ("trampoline") 4022 stored in the ROM 106 is 
executed when power is first supplied, which starts a program 4024 which installs the 
rest of the hardware resource dispatcher program 400 into memory, and starts it, 
10 passing as an argmnent a data structure (as described below) describing the system 
image configuration. 

The hardware resource dispatcher initialises a serial line which may be used for 
a system console. It tiien allocates memory space (an operating system enviroimient) 
for each operating system in turn, starting with the critical operating system. Ttie 
15 hardware resource dispatch^: therefore acts as a second level system kernel boot 
loadcX'. 

Each operating system kemel then goes through its own initiahsation phase, 
selecting the resources to be exclusive to that operating system within those remaining 
in the resoxirce allocation table 404, and starting its initial services and appUcations. 

20 Figure 6 illustrates an example of a memory address allocation forming the 

system image. A position within memory is allocated when tiie hardware resource 
dispatcher and operating systems are compiled. The set of these positions in memory 
defines the system image, shown in Figure 6. The system image comprises a first bank 
of memory 602 where the hardware resource dispatcher is located; a second bank of 

25 memory 604 where the real time operating system is located; a third bank of memory 
606 where the secondary operating system is located; and, in this embodiment, a fourth 
bank of memory 608 where the RAM disk containing a root file system of ttte 
secondary operating s>^tem (Linux) is located. 

This system image is stored in persistent storage (e.g. read only memory for a 

30 typical real time device such as a mobile telephone or PBX). The remaining banks of 
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memory are available to be allocated to each operating system as its environment, 
within which it can load and run applications. 

Allocation of Memory for Operating System Context 
5 Whilst being booted, each operating system then allocates a complementary 

piece of memory in order to meet the total size required by its own configuration. 
Once allocated to an operating system, banks of memory are managed using tlie 
ph3^ical memory management scheme of the operating systan itself. All ottier 
memory is ignored by the operating system. 

10 

Virtual Memory Allocation 

Each operating system is allocated separate virtual memory spaces, to make 
sure that operating systems caimot interfere with each other or with the hardware 
resource dispatcher. The User address spaces (i.e. ranges) and Supervisor address 
15 space (i.e. range) of each of the operating systems is each allocated a different 
memory management unit (MMu) context identifier (ID), which allow the 
differentiation of different virtual memory spaces having overlapping addresses. The 
MMUs context IDs are assigned to each operating system at the time it is compiled 
(step 314 of Figure 3). 

20 This solution avoids the need to flush translation cashes (TLBs) when the 

hardware resource dispatcher switches between diff^ent operating systCTis, which 
would take additional time. Instead, the switch over between different operating 
systems is accomplished by storing the MMU context IDs of the currently function 
operating system, and recalling the previously stored MMU context IDs of the 

25 switched two operating system. 

Allocation of Input/Output Devices 

As indicated above, the allocation table 404 indicates which devices axe 
allocated uniquely to each operating system. In addition, table 404 indicates which 
30 input/output resources (Direct Memory Access (DMA) devices, input/output ports, 
internets and so on) are allocated exclusively to such devices, thus allowing a direct 
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use of these resoxirces without any conflict. Typically, many devices are duplicated, so 
it is possible to reduce potential conflicts substantially in this way. 

The distribution is based on the operating system configuration scheme (for 
example, in the case of C5, the devices specified in the device tree). They are 
5 allocated to operating systems at boot time, and in order of booting, so tiiat tiie critical 
operating system has first choice of the available devices in the table 404 and the 
secondary operating systems in turn receive their allocation in what remains. As each 
operating system initialised, it detects the presence of these devices and uses its native 
drivers for them without interaction from the hardware resource dispatcher. 

10 

"Hot" Reboot of Secondary Operating System 

According to the present embodiments, it is possible to reboot a secondary 
operating system (for example because of a crash) whilst other operating systems 
continue to run. Because of the separation of system resources, a crash in the 
15 secondary operating system does not interfere witii the ongoing operation of the 
critical operating system (or other secondary operating systems) and the rebooting of 
tiiat secondary operating system does not do so either. 

In the embodiment, the system "stop" and "start" tr^ caUs to the hardware 
resource dispatcher assist in shuttmg down and restarting the secondary operating 
20 systems firom within the critical operating system. Additionally, the hardware resource 
dispatcher saves a copy of the original sj^em image, at boot time, in persistent 
memory within the hardware resource dispatcher allocated memory. As an example, 
hot restart in this embodiment is managed as follows: 

At the time of initially booting up, the hardware resource dispatcher saves a 
25 copy of the secondary operating systems memory image. 

The critical operating system includes a software watchdog driver routine for 
periodically monitoring the functioning of the secondary operating systCTsis (for 
example, by setting a timeout and waiting for an event triggered by a peer driver 
running in the secondary operating systems so as to check for their continued 
30 operation). 
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If the critical operating system detects that the secondary operating system has 
failed or stopped, it triggers "stop" and then "start" trap calls (of the secondary 
operating system) to the hardware resource dispatcher. 

The hardware resource dispatcher then restores the saved copy of the secondary 
5 operating system image, and reboots it from memory to restart. It was found that, on 
tests of an embodiment, the Linux secondary operating system could be rebooted 
within a few seconds from locking up. 

In other respects, the hot restart builds upon that available in the Chorus operating 
system, as described for example in: 
10 "Fast Error Recovery in CHORUS/OS. The Hot-Restart Technology" . 

Abrossimov, F. Hermann. J.C. Hugly, et al, Choms Systems Inc. Technical Report, 
August 1996, 14p. available from: 

http://www.jaluna.com/developer/papers/CSI-TR-96-34.pdf 

1 S Run-time Operation 

The operation of the embodiment after installation and booting will now be 
described in greater detail. 

Having been booted and initialised, the real time operating system is running 
one or more applications 207 (for example a UDP/IP stack - UDP/IP stands for 
20 Universal Datagram Protocol/Intemet Protocol) and the secondary operating system is 
nmning several ^plications 208a, 208b (for example a word processor and a 
spreadsheet). The real time operating system microkernel 201 and the secondary 
operating system kernel 202 communicate with the hardware resource dispatcher 
through the hardware resource dispatcher interface which comprises: 
25 •a data structure representing the operating system context (i,e. the set of state 
variables which need to be saved and restored in order to switch to the operating 
s}^tem), and the hardware repository; 

• the set of functions which execute in the operating system environment; and 

• the set of tr^ call routines which execute in the hardware resource dispatcher 
30 environment 
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If neither operating system requires processor time (for example, both have 
reached "wait" states) then the hardware resource dispatcher 400 switches to the 
critical operating system's idle thread, in which it waits an interrupt or evCTit. Thus, 
intCTiq)ts can be processed immediately by the critical operating system's servicing 
5 routines, without needing to switch to the critical operating system first. 

At some point, an inteirupt or event will occur. For example, a packet may be 
received at a data port, causing an interrupt to allow it to be processed by the real time 
operating system executing the UDP/IP stack. Altematively, a user may manipulate a 
keyboard or mouse, causing an interrupt to operate the GUI of the second operating 
10 system 202 for interaction with the word processing appHcation 208. Altematively, 
the system clock may indicate that a predetermined time has elapsed, and that an 
application should conmience re-execution, or an operating system function should 
execute. 

The critical operating system servicing routine then services tiie interrupt, as 
1 5 described below. 

Interrupt and Event Handling 

If not already in the critical operating system, the hardware resource dispatcher 
interrupt handler 412 calls the operating system switcher 408 to switch to the critical 

20 operating system, and then the interrupt handler routine 412 to call an intermpt service 
routine (ISR) in the critical operating system 201. If the interrupt is intended for the 
critical operating system, either because it is from a device uniquely assigned to the 
critical operating system or because it is from a shared device and has a certain 
predetermined value, the critical operating system ISR takes the action necessary to 

25 handle the interrupt. If not, control is passed back to the hardware resource dispatcher. 

Critical to Secondary Operating Systems Switch 

Referring to Figure 7, for this example, the system is executing a thread 702 of 
an application 207a running on the critical operating system 201 . 
30 If an interrupt occurs, a critical operating system interrupt service routine 704 

performs interrupt servicing. On termination, control passes back to the thread 702 
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and any others executed by the scheduler of the critical operating system 201. When 
processing of all threads is complete, the critical operating system has fiboished 
executing, it schedules its "idle" thread. Accordingly the "idle" tr^ routine in the 
critical operating system issues an "idle" trap call to tiie hardware resource dispatcher 
5 400. The hardware resource dispatcher then executes a routine which does the 
following: 

• If the intemipt handler 412 currently has some stored virtual interrupts, these 
are forwarded by the interrupt handler 412 to the secondary operating system. 

• The hardware resource dispatcher operatiag system schedvder 410 selects the 
10 secondary operating system 202 to execute. The OS switcher 408 tiien saves 

the current context (typically, processor MMU and status registers, instruction 
and stack pointers) in the critical OS context storage area 706. It then retrieves 
the stored execution context 708 for the secondary operating system 202, and 
writes them to the registers concemed. 
15 •If there are virtual interrupts for the secondary OS concemed, the interrapt 

handler 412 calls the relevant interrapt service routine 710 witiiin the secondary 
operating system, which services the intemq)t and then, on completion, reverts 
to the execution of a thread 712 of the secondary operating system wh©re it left 
ofiF. 

20 If the interrapt handler 412 currently has no pending interrapts, then the 

hardware resource dispatcher operating switcher 408 causes the secondary operating 
system to recommence execution where it left off, using the stored program counter 
value within the restored operating system context, in this case at the thread 712. 

Thus, after the critical operating system 201 has performed some fimction 

25 (either servicing its own appUcations or services, or servicing an intemq)t intended for 
anotiier operating system), the hardware resource dispatcher passes control back to the 
next secondary operating system 202, as determined by the scheduler 410. 

Secondary to Critical Operating System Switch on interrapt 

30 Referring to Figure 8, the process of transferring from the secondary operating 

system to the critical operating system will now be disclosed. Li this case, the system 
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is executing a thread 712 of an application 208a running on the critical operating 
system 202. 

When a hardware interrupt occurs, the hardware resource dispatcher starts tiie 
OS switcher, to save the secondary operating system context in the context storage 
area 708. It then switches to the primary operating system 201, restoring the values of 
state variables from the context storage area 706, and calls the interrupt service routine 
704 of the primary operating system 201. After servicing the interrupt, the scheduler 
of the primary operating system 201 may pass control back from the ISR 704 to any 
thread 704 which was previously executmg (or thread to be executed). 

When the ISR and all threads are processed, the primary operating system 201 
passes control back to the hardware resource dispatcher, which switches from the 
primary operating system 201 (saving the state variables in the context storage 706) 
and switches to a selected secondary operating system 201 (retrieving the state 
variables from the context storage 708), in the manner discussed with reference to 
Figure 7 above. 

Inter-operating system communications - virtual bus 418 

The virtual bus routine cooperates with the virtual bus drivers in each operating 
system. It ©cnulates a physical bus comiecting the operating systems, similar to 
Compact PCI (cPCI) boards plugged into a cPCI backplane. Each operating system is 
provided with a drivOT routme for the virtual bus bridge device on fliis virtual bus, 
allowing the operatiug systems and their applications to communicate by any desired 
protocol, from raw data transfer to a frill IP protocol stack. 

The hardware resource dispatcher virtual bus is based on shared memory and 
system cross interrupts principles already discussed above. In detail, the virtual bus 
routine 418 emulates the C5 buscom DDI: syscom which defines virtual bus bridge 
shared devices, allowing the export (sharing) of motnory across the virtual bus and 
triggering of cross-interrupts into other operating systems. 

Each virtual bus driver, in each secondary operating system, creates such a 
virtual bus bridge in the hardware resoiirce dispatcher hardware repository at startup 
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time. By doing so, it exports (shares) a region of its private memory, and provides a 
way to raise interrupts within its hosting system. 

Thus, a virtual bus driver of a first operating system sends data to a second 
operating system by: 

5 • writing into the memory exported by a peer virtual bus driver of the second 

operating system, and then; 
• triggering a cross-interrupt to notify that data are available to the peer bus 

drivCT in the second operating system. 
In the reverse (incoming) direction, the virtual bus driver propagates incoming data 
10 up-stream (for use by the application or routine for which it is intended) when 
receiving a cross-interrupt indicating that such data have been stored in its own 
exported memory region. 

Referring to Figure 9a, an appUcation 208a which is to communicate with 
another 208b running on the same operating system 202 can do so through that 
15 operating system. An application 207b running on one operating system 201 which is 
to communicate with another 208b running on a different operating system 202 does so 
by writing data to the virtual bus using the API of its operating system, which uses the 
virtual bus driver routine to pass the data to the other operating system 202, which 
propagates it from its virtual bus driver to the application 208b. 
20 Referring to Figure 9b, the changes necessary to migrate this arrangement to 

one in which the first and second operating systems run on different computers 100, 
101 are small; it is merely necessary to change the drivers used by the operating 
systems, so that they use drivers for a real bus 103 rather than the virtual bus drivers. 
The system is therefore made more independent of the hardware on which it operates. 
25 Commimication across the hardware resource dispatcher virtual bus is available 

to applications, but can also be used internally by the operating system kemels, so that 
they can cooperate in the implementation of services distributed among multiple 
operating systems. "Smart" distributed services of tiiis kind include software watchdog 
used for system hot restart (discussed above), or a distributed network protocol stack. 

30 

Debugging 
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Li a preferred embodiment, the hardware resource dispatcher has a second 
mode of operation, in which it acts as a debugging agent. 

According to this raibodiment, in the second mode, the hardware resource 
dispatcher can communicate via a serial communications line with debugging software 

5 tools running on another machine (the **host" machine). 

Such debugging tools provide a high level gr^hical user interface (GUT) to 
remotely control the hardware resource dispatcher. The hardware resource dispatcher 
virtualised exception mechanism is used to intercept defined exceptions. The user can 
then configure and control how the hardware resource dispatcher behaves in case of 

10 processor exceptions, and also display machine and system states, to enable diagnosis 
of code or other system errors or problems. 

The user can select one or more such processor exceptions as the basis for a 
trap call from an operating system to the hardware resource dispatcher. On the basis of 
the selected exception, when the or each exception occurs during execution, the 

IS operating system is stopped, and executes the trap caU to the hardware resource 
dispatcher, which then saves the current context and enables interaction with the 
debugging tools on the host. The user can then cause the display of the current states 
of the state variables (such as the stack pointers, program and address countCTs) and/or 
the content of selected block of memory. The user can specify either that a given type 

20 of exception should be trapped in a specific operating system to be debugged, or that 
they should be trapped whenever they occur, in any operating system. In response, the 
trap call is implemented in just one, or in aU, operating systems. The user can also 
specify if a given type of exception is to be normally forwarded to the system when 
restarting execution or simply ignored. 

25 Because the hardware resource dispatcher executes in its own environment, it is 

able to debug much more of an operating system than could be done from within that 
system. Importantiy, no code is shared between the hardware resource dispatcher 
acting as a debug agent and the sytems being debugged. This allows, for example, the 
debugging of even kernel low level code such as exception vectors or interrupt service 

30 routines. 
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Some other aspects of the overall (host/target) debuggmg architecture 
according to this embodiment are similar to those for the Chorus and C5 debugging 
systems, described in the docimient "C5 1.0 Debugging Guide" pubUshed by Jaluna, 
and available at: 

5 http://wwwjaluna.com/doc/c5/html/DebugGuide/bookl .html 

Secure Architecture 

It will be clear that the embodiments described above give a firm basis for a 
seciire architecture. This is because the secondary operating system, on which a user 

10 will typically run insecure applications, is insulated from specified system resources, 
and accesses them only through ttie hardware resource despatcher (and the drivers of 
the primary operating system). Thus, security appUcations can be run on the primary 
operating system which, for example, perform encryption/decryption; allow access to 
encrypted files; manage, store and supply passwords and other access information; 

15 manage and log access and reproduction of copyright material. AppUcations running 
on the secondary operating system cannot access system resources which are not 
allocated to that operating system, and where the operating systems run in different 
memory contexts (i.e. use different addressing pointers to dijl^erent spaces) 
applications running on the secondary operating system caimot be used to interfere 

20 with those operating on the primary system so as to weaken the security of its 
operations. 

This section describes the invention on PowerPC ^TPC") architecture, as an 
example of a Reduced Instruction Set Computer (RISC). For a background 
understanding of this well known architecture, the following are incorporated by 
25 reference: 

'•PowerPC Microprocessor Family: The Programming Environments for 32-Bit 
Microprocessors - Software Reference Manual" (pubUshed by IBM Inc.), Publication 
Number: G522-0290-01 Revision Date: 02/21/00 

available for download from: 

30 http://v(nvw.ibin.coni/chips/techhV^^ 
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In the following, the hardware resource despatcher is described (m a non- 
limiting sense) as a nanokemel. This section focuses on PPC specifiic aspects of the 
nanokemel implementation, in particular, on the nanokemel executive which is the 
comer stone of the nanokemel envirorment. 

This section describes how the PowerPC processor architecture is used in order 
to implement flie nanokemel executive which is capable to run multiple independent 
operatmg systems concurrently sharing the central and co-processing units (CPU and 
FPU) as well as the memory management unit (MMU) across these operating systems. 

It also describes how the nanokemel executive handles the hardware interrupts. 
In particular, it describes the mechanism used to intercept and forward hardware 
interrupts toward the primary operating system and the software intermpts mechanism 
provided to the secondary operatmg systems. 

Note that in this document we assimie that the nanokemel is running on a 
uniprocessor computer and therefore aspects related to the symmetrical multi- 
processor. (SMP) architecture is not addressed here. 

Overview 

Virtual Address Spaces 

On PowerPC architecture the nanokemel always runs in the effective (physical) 
20 address space. In other words, the MMU is always disabled, and the processor is 
running in real mode when nanokemel code is executed. 

In this description the memory context term designates a PowerPC virtual 
address translation context: 

^ a set of 16 virtual segment identifiers (VSID), specified 
25 in the segment registers 

js^ a set of page table entries (PTE) 

js$ a set of block address translations, specified in the BAT 
registers. 

Typically, an operatmg system siq>porting user mode processes creates multiple 
30 memory contexts (one per user process) in order to be able to handle private user 
virtual address spaces. The kernel changes the mraiory context each time it switches 



10 



wo 2005/033928 



PCT/IB2004/003344 



28 

from one user process to another. Additionally the operating system kernel also 
handles a unique supervisor address space. User and supervisor virtual addresses can 
overlap on PowerPC architecture. 

The supervisor address space mappings may be either static or dynamic. The 
S static mapping is created at system initialization time and it typically maps (entirely or 
partially) available physical memory. Such mapping is also called the one-to-one or 
kemel virtual (KV) mapping and typically use the PowerPC Block Address Translation 
mechanism (BAT). In particular, the KV mapping usually covers the kemel code, data 
and bss sections. Dynamic mappings are created at run time in order to access 
10 dynamically loaded kemel modules or dynamically allocated (non contiguous) 
memory chunks. 

Three kinds of memory context are distinguished in the nanokemel 
environment: primary, secondaiy and nanokemel. 

The primary memory context is a memory context ciirrently used by the 
IS primary kemel. Note that, in case the primary operating system supports user address 
spaces, there might be multiple memory contexts used by the primary kemel but, as 
was akeady mentioned above, the supervisor address space is unique. Because the 
nanokemel does not care about user mappings, the primary memory context is unique 
from the nanokemel perspective and it consists in static and dynamic supervisor 
20 mappings established by the primary kemel. 

The secondary memory context is a memory context currently used by the 
secondary kernel. Once more, in case the secondary operating system supports user 
address spaces, there might be multiple memory contexts used by the secondary kemel 
but there is only one supervisor address space. Thixs, the secondary memory context is 
25 unique from the nanokemel perspective (for a given secondary kemel) and consists in 
its supervisor memory context. 

The nanokemel itself do not really use a memory context as defined above but 
rather the PowerPC processor effective (physical) address space. However, the 
nanokemel address space is different from all other memory contexts and therefore can 
30 be considered as a specific one. 
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The nanokemel memory context is mainly used to execute the nanokemel code 
when a secondary kernel is preempted by an interrupt, trap or exception event handled 
by the nanokemel, for example, in order to perform an I/O operation to the nanokemel 
console. The nanokemel memory context is also used as an intermediate address space 
5 allowing to switch from a secondary execution environment to the primary one and 
vise versa. Note that because the PowerPC processor switches to real execution mode 
for exertion processing, it is a natural and efficient ^proach to use the processor 
physical address space as the nanokemel memory context. 

Figure 1 shows an example of the primary and a secondary virtual address 

10 space as well as the nanokemel physical address space. 

In this example the physical memory size is 128 megabytes. The primary 
kemel uses the trivial one-to-one (KV) mapping starting from zero (like C5 
microkemel) and the secondary kemel uses a shifted one-to-one (KV) mapping starting 
from OxcOOOOOOO (like Linux kemel). 

15 Figure 2 shows an example of how the memory context is switching in time. 

Initially, a secondary operating system is running in a secondary memory context. At 
to time, the current secondary kemel tr^s to the nanokemel in order to output a 
character to the nanokemel console. This trap switches the current memory context to 
tiie nanokmiel one. During the [tO,tl\ p^iod, the nanokemel (r unning in the 

20 nanokemel memory context) prints out a character to tihe nanokemel console. At tl 
time, the nanokemel returns to the secondary kemel switching back to the secondary 
memory context. At t2 time an intermpt occurs while running the secondary operating 
system. The intermpt switches the current memory context to the nanokemel one and 
invokes the nanokemel intermpt handler. In order to forward the intermpt to the 

25 primary kemel, the nanokemel switches from the nanokemel memory context to the 
primary one and invokes the primary intermpt handler at t3 time. During the intermpt 
request processing, at t4 time, the primary kemel traps to the nanokemel in order to 
output a character on the nanokemel console. 

At t5 time, the nanokemel returns from the putchar trap call to the primary 

30 kemel which continue the intermpt request processing until the t6 time. At this 
moment, the prin[iary kemel returns from the interrupt handler and the nanokemel 
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switches back to the interrupted secondary operating system in order to continue its 
execution. Such a switch starts in the primary memory context and, going flirough flie 
intermediate nanokemel context, finally ends up in the secondary memory context at t7 
time. 

Nanokernel Invocation and Preemption 

The nanokemel is invoked either explicitly through a trap or implicitly through 
an interrupt/exception handler. In the former case, we say that an operating system 
kernel invokes the nanokemel. In the latter case, we say that the nanokemel preempts 
an operating system. It is important to underline that the nanokemel is always invoked 
jQrom the privileged code running in the supervisor address space. On the other hand, 
the nanokanel may preempt the kernel itself as well as an user process rmming under 
kernel control. In fact, it is undesirable, and counterintuitive, to permit the nanokemel 
to preempt the primary kernel, and it does not do so except for floating point unit 
sharing (as discussed below) — the saving in switching time realised by lazy floating 
point sharing outweighs the drawback of occasional pre-emption. 

Once the system is booted, the nanokemel is activated first and it starts 
execution- of the primary and secondary kernels. Once the initialization phase is done, 
the nanokemel plays a passive role. This means that the code executed in the 
nanokemel is driven by the primary and secondary kernels explicitly invoking the 
nanokemel (by trap) or by externally generated synchronous (i.e., exceptions) and 
asynchronous (i.e., intemipts) events. 

On PowerPC architecture, mechanisms used for the nanokemel invocation and 
preemption are the same for primary and secondary operating systems. In terais of 
execution environment, the nanokemel is quite separate firom the primary and 
secondary kernel as it runs in PowerPC real mode. It uses a"null" memory context 
(physical address space) and a different supervisor stack. There is a barrier between the 
operating systems (MMU enabled) and the nanokemel (MMU disabled) that provides 
some protections against the kernel malfunction. Note however that such a protection 
is not absolute because each kernel stiU runs privileged code in supervisor address 
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space and therefore a secondary kernel is still able to crash the primary kernel as well 
as the nanokemel. 

Nanokernel Invocation 

The primary and secondary kemels invoke the nanokemel using a trap call 
mechanism. The PowerPC sc and trap instructions are not used to avoid the 
interception of the program and system call exceptions by the nanokemel. This would 
introduce performance overhead in the processing of these exceptions that are 
frequently used by an operating system kernel. 

Instead, some exception vectors currently unused by PowerPC processor 
implementations, are dedicated to nanokemel trap calls (avoiding using the operating 
system trap calls). These exception vectors are called through a small software routine, 
running in the primary or secondary kernel execution environment, that puts the 
PowerPC processor in the same state as when a real exception occurs, and jumps to the 
appropriate vector using the rft instmction. The kemels are appropriately modified. In 
other words, the nanokemel invocation mechanism extends the existing PowerPC 
exception set by simulating software exceptions triggered under control of the primary 
or secondary kraiels. 

The following exception vectors are used for nanokemel trap calls: 

0x2000idle, invoke nanokemel scheduler 

0x2100invoke nanokemel debug agent to stop execution 

Qx2200invoke nanokemel console lO services 

0x2300trigger a system cross-interrupt 

Qx2400restart system 

0x2500process pending virtual exception, if any (secondary only) 
0x2600halt system (secondary only) 

Nanokemel invocation is made with following convention, and processor state: 
js$ translation are disabled (switch to real mode) 
js^ interrupts are disabled at processor level 
js^ caller r20 is saved in sprgO 
je^ cdHer r21 is saved in sprgl 
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^ caller msr is saved in r20 

^ caller return address (next instruction to execute) is 
stored in r21 

jss sub-trap number is stored in r9 (for multi-function 

5 traps) 



Nanokernel Preemption 

The nanokemel preempts the operating systems by intercepting the PowerPC 
exception / interrupt vectors. The PowctPC architecture does not provide any 
10 mechanism (like a base register) to configure the address where the exception vectors 
are located, but enforces them to be located at the beginning of the physical address 
space (at 0x00000000 in RAM) or in the jBrst page of the last megabyte of the physical 
address space (at OxfEfOOOOO usually in ROM). Therefore, the nanokemel owns the 
PowerPC real exception vectors and uses an array of indirect function pointers (an 
15 exception handler table) to call the native system handler or to intercept tiie exception 
and execute a nanokemel handler insiead. When an operating system is preempted by 
the nanokemel the processor state is automatically changed by the taken exception 
(real mode). Additionnally, the nanokemel handler may switch to another memory 
context and supervisor stack, to execute a task in its own environment, or to directly 
20 switch to another operating system. 

When the nanokemel just forward the exception to the native kernel handler it 
has to modify the regist^ contents in order to implement an indirect call without 
loosing the processor state. Thus the nanokemel apply the following convention about 
register usage when calling a kernel exception handler (we only mention what differs 
25 from the state at exception entry): 

js^ r20 is saved into sprgO scratch register 
^ r21 is saved into sprgl scratch register 
jg^ Ir is saved into r20 register 

j8$t21 is loaded with the exception table index (excq)tion 
30 number * 4) 
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^ It is loaded with the address of the kernel exception 
handler 
Primary Preemption 

The nanokemel preempts the primary operating system only in rare cases. 
5 Typically, only tiie exceptions used to manage unavailable co-processing units are 
intercepted (FPU and AltiVec unavailable exceptions). These exertions are used by 
the nanokemel to handle the unit sharing between kernels in a lazy fashion as 
described later in this document. 

Secondary Preemption 

hi addition to the co-processing exceptions, a secondary operating system is 
also preempted when an interrupt (asynchronous exception) occurs. In such a case, the 
interrupt vector is intercepted by corresponding nanokemel handler (installed in 
secondary exception table). The nanokemel then switches to the primary memory 
context and call the associated primary kernel handler for this intemipt (installed in the 
primarj' exception handler table). 



10 



15 



Kernel Context 

The nanokemel data can be split on two categories: the global and per-kemel 
20 data. The global data keeps the global nanokemel state (e.g., the nanokemel memory 
context) while the per-kemel data keeps a state associated to a given primary or 
secondary kemel. The per-kemel data is also called the kernel context. 



The kemel context consists in two parts: visible and hidden. The visible part is 
public and takes a part in the nanokemel interface. This part of the kemel context is 
described in detail in further sections related to the nanokemel interface. The hidden 
part is not visible to kernels and used internally by the nanokemel executive. 

Nanokemel Executive Interface 

This chapter describes the nanokemel executive interface exported to the 
primary and secondary kernels. Such an interface consists in a data structure shared 
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between a kernel and the nanokemel (i.e., visible kernel context) as well as the 
nanokemel methods. 

Visible Kernel Context 

5 Figure 3 illustrates Ihe visible part of the kernel context. 

Note that, in the visible part of the kernel context, all references are made 
through physical addresses. A kemel has to convert such a physical address to the 
virtual one (from the KV mapping) in order to access the referenced object. The 
picture shows a configuration with only two kernels: primary and secondary. 

10 The hdlsO field is an array of pointers to the kemel exception handlers. This 

array is indexed by the exception number. Each entry in this array may be set directly 
to the kemel native exception handler or to a nanokemel handler in order to intercept 
the associated exception. In the later case the kemel native exception handler pointer 
is located in the VEXhdlsO field. 

15 The pending KEXand enabled KEZ fields reflect the current state of the virtual 

exceptions. Note that these fields are meaningless for the primary context because the 
primary kemel exceptions are not virtualized by the nanokemel. The virtualized 
exceptions mechanism is described in detail fiirther in this document together with the 
secondary kemel execution model. 

20 The boot info field points to a global boot information structure. This field is 

read-only. 

Such a data stmcture contains various processor information (frequencies) as 
well as a pointer to any firmware passed argument. 

The cmd_line starts and size parameters points to the boot command line 
25 specifying the boot time parameters. Such parameters are given to the boot loader or 
passed through the nanokemel environment. The command line is kemel specific and 
it is located in the kemel context. The nanokemel parses the initial (multi-system) 
command line in order to create kemel specific command lines containing only 
parameters related to the corresponding kemel. 
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The RAM info field points to the RAM description table. This field is read-only. 
The RAM description table is a global data structure shared by all kernels* It describes 
how the RAM resource is distributed across the kernels. 

The dev info field points to the list of virtual devices abstracted by the 
5 nanokemel. This field is read-only for a secondary kernel and read-write for the 
primary one. The devices list is global and it is shared by all kernels. Each virtual 
device in flie list is represented by a data structure specified by the nanokraiel. This 
data structure is typically accessed by both primary and secondary peer drivers using 
rules defined by the nanokemel. The primary peer driver plays a server role supporting 

10 the virtual device while the secondary peer driver plaj^ a client role using the virtual 
device instead of the real one. This Ust is created (and modified) by the primary kernel 
only. A secondary kemel is only allowed to browse this Ust. 

The pending XIRQ field specifies pendiag cross interrupts. This field is not 
used by the nanokemel itself. It is hosted by the context structure in order to assist to 

15 the primary and secondary kernels in the cross interrupts exchange. There is only one 
processor exception dedicated to the cross intermpt denvery. The pending XIRQ field 
allows to extend the number of cross interrupts up to 32 (one bit per cross interrupt 
source). A cross intermpt bit is set by the source kemel (i.e., ttie konel which sends 
cross intermpt) and it is reset by tiie destination kemel (i.e., the kemel which receives 

20 the cross interrupt). 

The ID field contains a unique kemel identifier. This field is read only. 
Identifier 0 is assigned to the nanokemel itself and identifier 1 is assigned to the 
primary kemel. The kemel identifier designates the kemel in the nanokemel interface. 
For example, the kemel identifier is used to tag resources assigned to a given kemel 

25 (e.g., memory chunks in the RAM description table). 

The running field points to a system identifiers bit field specifying the state of 
corresponding kernels: ruiming (1) or halted (0). This bit field is read only. The 
nanokemel sets a bit before launching the associated kemel and clears it once this 
kemel is halted. When a kemel is restarted, its running bit is first cleared and then set. 

30 Any kemel is able to analyze the running bit field in order to find out all ruiming peer 
kernels. Note that the running bit of the primary konel is always set. 
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The ctx root^ and last fields point to respectively the first (nanokemel itseLQ 
and last valid kernel contexts. The ctx size field specifies the whole size of a kernel 
context structure including the hidden part. These fields together provide required 
information to manage the kemel contexts. 
5 The shared memory field points to a pool of shared memory. It is used, mainly 

by the primary kemel to allocate memory to store data shared between all the kernels. 

Nanokernel Methods 

The nanokemel provides two groups of methods: the console I/O operations 
10 and the executive operations. The console I/O group allows a kemel to send/receive 
characters to/fi:om the nanokemel console serial hne. This docummt does not specially 
address the console I/O methods which are more or less generic but rather it is focused 
on the executive methods which are PowerPC architecture specific. 

Install CPU exception handler 

Instead of installing exception vector code directly into the PowerPC exception 
vectors, a kemel has to invoke this nanokemel method to attach a handler to a given 
processor exception. The exception number and physical address of the kemel handler 
code are passed as arguments. The exception number is used to index the hdlsf] 
exception handler table in the kemel context, where the kemel handler pointer is 
stored. 

The nanokemel can later use the table entry value to directly raise 
corresponding exception to the kemel with the minimum overhead of an additional 
indirect call. This metibod is to be used for exceptions that are directly processed by the 
kernels (not intercepted by the nanokemel). 

InstaU virtualized exception handler (VE^Q 

This method is used to attach a kemel exception handler to an exception 
virtualized by the nanokemel. Such an exception is either a virtual exception 
30 corresponding to a real Pow^PC exception intercepted and deferred by the 
nanokemel, or a logical event raised by the nanokemel. 
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The VEX number and physical address of the kernel handler code are passed as 
arguments. The VEX number is used to index the vexHdls/J virtualized exception 
handler table in the kemel context, where the kernel handler pointer is stored. 

Idle 

The nanokemel provides an idle method which has to be called by a kemel 
wifliin an idle loop. The idle method informs the nanokemel that the calling kemel has 
nothing to do until the next intermpt 

The idle method invocation results in a system switch to the next ready to run 
secondary kemel (if any) or in the return from the primary idle method when all 
secondary kernels are idle. The idle method has no parameter. 

Restart 

The nanokemel provides a restart method which can be called by the primary 
as well as by a secondary kemel in order to restart a secondary kemel. 

The method parameter specifies identifier of the kemel being restarted. The 
nanokemel stops the kemel execution, restores the kemel image from its copy and 
finally starts tiie kemel execution at the initial entry point. 

Note that a secondary kemel can reboot itself by calling the restart trap with its 
own identifier. 

Secondary Halt 

The halt trap is provided by the nanokemel to a secondary kernel. Such a tr^ is 
called by a secondary kemel when it is halted. The nanokemel puts the caller kemel 
25 into a non running state in order to avoid this kemel being switched in by the 
nanokemel scheduler. 

A stopped kemel can be started again by the restart nanokemel method 
described above. 
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Primary Execation Environment 

Basically, the primary kernel is executing in the native execution environment. 
The nanokemel implementation on PowerPC processor tries to nGonimize impact of the 
nanokemel environment to the primary operating system characteristics (performance, 
5 interrupt latency, preemption latency). Because the primary operating system is 
typically a real-time operating system, it is important to keep the primary kernel 
behavior unchanged even if other (secondary) operating systems are ruxming 
concurrently on the same processor. 

10 Initialization 

The nanokemel is started first by the boot loader with disabled MMU, i.e., in 
the physical space. Basically, the nanokemel initialization code installs the primary 
memory bank (containing the primary kernel code/data^ss sections) and the other 
banks in the physical memory and jumps to the primary entry point. 

15 Before jimiping to the primary kernel, the nanokemel initializes the primary 

kernel context by calling a system specific fimction. This function shoidd at least set in 
the hidden part of the kernel context.: the srrO register image to the kernel entry point 
and the srrl register image to the initial value expected by the system kernel. 

All entries in the exception handler table (hdlsff field of the kernel context) 

20 point to the nanokemel debug agent entry point, except for the co-processing (FPU) 
unit exceptions. This ensure that any unexpected early exception will stop execution. 

The nanokemel initialization code is executed using a sqparate static 
nanokemel stack located in the data section. When jxmiping to the primary kernel, this 
stack is still vaUd. Despite of that, the primary kernel should switch to its own stack as 

25 soon as possible and should never use this nanok^nel stack in the fiiture. The 
nanokemel stack is used not only at initialization phase but also at run time in order to 
handle secondary invocations and preemptions as described in the next chapter. 

When jumping to the primary kernel, the r3 register points to the kernel context 
and the msr register is loaded with the sirl image set in the kernel context Typically, 

30 the processor interrupts are disabled at the beginning of a primary initialization phase. 
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The primary kernel usually enables interrupts once a critical initialization phase is 
done. 

During the initialization phase, the primary kernel typically invokes the 
nanokemel methods in order to attach handlers to the processor and virtualized 
5 exceptions. Finally the primary kernel enters in the idle loop and invokes the 
nanokemel idle method. 

When the idle method is called first time, the nanokemel considers that the 
primary kernel has fully initialized its execution environment and it proceeds to the 
post initialization phase. 
10 In such a post initiaUzation phase, the nanokemel initializes the secondary 

kernel contexts as described in the next chapter. Once the post initialization is done, 
the nanokemel calls the scheduler in order to either switch to a ready to run secondary 
kernel or return Jfrom the primary idle method if all secondary kemels are idle. 

The nanokemel requires the primary kernel to initialize the globally shared data 
15 stmctures: the RAM descriptor and the virtual devices list. Such an initialization has to 
be done before the idle method is called. This reqairement is natural because beyond 
this moment a secondary kernel can access the globally shared data stmctures. 

In particular, the primary kernel is in charge to detect the physical memory 
available on the board and to register free physical memory chunks in the RAM 
20 descriptor. 

According to the primary Board Support Package (BSP), the primary kemel 
should start nanokemel aware drivers which, in turn, should populate the virtual 
devices list. Such virtual devices are provided to secondary kemels and therefore they 
should be created before the first secondary kemel is started. 

25 

Intercepted Exceptions 

Basically, the nanokemel does not intercept exceptions which occur when the 
primary operating system is rurming on the processor. All programming exceptions, 
traps and intermpts are handled by native primary handlers. The primary low-level 
30 handlers just need to be modified in order to comply witii the nanokemel exception 
handler calling convention for PowerPC architecture. 
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An exception from the above rule is programming exceptions related to the co- 
processing units emulation: 

js£ the floating point unavailable exception (FPU) 
the vector unit imavailable exception (AltiVec VU) 
5 These exceptions are used by the nanokemel to implement a lazy mechanism 

of co-processing units sharing as described further in this document. 

Another special case is a debug agent which could be embedded in tiie 
nanokemel ia order to provide a host based remote system debugging of the primary 
operating system. In this case, the debug agent usually intercepts some synchronous 
10 exceptions related either to debug features (e.g., single iustruction trace) or to program 
errors (e.g., page fault). Such a debug agent design is however out of scope of this 
document. 

Forwarded Interrupts 

IS When an iutermpt occurs while a secondary operating system is runmng on the 

processor, the interrupt is forwarded to the primary operating system. Such an interrupt 
forwarding process goes through the following major steps: 

jg^the interrupt is intercepted by the nanokemel 
(corresponding entry of the hdlsQ table in secondary kernel 
20 context points to a nanokemel handler); 

js^ execution of the preempted secondary kernel is 
suspended and the nanokemel switches to the primary execution 
environment; 

^ the nanokemel triggers the interrapt to the primary 
25 kanel (branch to corresponding entry of the hdlsQ table in 

primary kemel context). 

In such a way the corresponding primary low-level interrupt handler is invoked 
(in the primary execution environment) in order to process the intermpt. Once flie 
30 intermpt is processed, the primary kemel returns to the nanokemel executing a rfi 
instruction. 



wo 2005/033928 



PCT/IB2004/003344 



41 

After returning from the primary interrupt handler, the nanokemel calls the 
scheduler in order to determine the next secondary operating system to run. Note that 
the preempted secondary system would not necessary he continued after interrupt. 
Another (higher priority) secondary system may become ready to run because of tiie 
interrupt. 

Secondary Execution Environment 

Basically, the secondary kernel execution environment is quite closed to the 
native one except for the interrupts management. The nanokemel environment 
modifies the native mechanism of the interrupts management in order to make a 
secondary operating system fully preemptable. A secondary kernel ported to the 
nanokemel architecture no more disables interrupts at processor level but rather uses a 
software intemipts masking mechanism provided by the nanokemel (i.e., virtual 
exceptions). Interrupts are no more directly processed by such a secondary kemel, but 
rather they are intercepted by the nanokemel, forwarded to the primary kernel and only 
tiien optionally processed by the secondary kernel in a deferred way. 

Initialization 

The nanokemel installs the secondary memory banks at initialization time 
together with primary banks. On the other hand, the final initialization of a secondary 
kernel, in particular the kernel context setup, is deferred until the post initialization 
phase. 

At this phase, the nanokemel allocates memory to keep a copy of secondary 
memory banks. Such a copy is then used to restore the initial image of secondary 
system at restart time. The secondary system restart is however optional and it might 
be disabled in order to reduce the physical monory consumption. 

The secondary kernel context is setup by calling a system specific routine that 
initializes the context according to what is expected by the system entry point. All 
entries in the exception handler table {hdlsO field of the kernel context) point to the 
nanokemel debug agent entry point, except for the co-processing (FPU) unit and flie 
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interrupt entries. These ones point to nanokemel specific handlers used to intercept 
corresponding secondary exceptions as described in the next section. 

The nanokemel launches a secondary kernel by switching to the initial 
execution context previously setup by ttie system specific routine in the kernel context 
hidd^ part. 

Analogous to the primary kemel, the kemel context physical address is passed 
as an argument (passiug convention is system specific). On the other hand, unlike the 
primary kemel, the interrapts are enabled at processor level (MSR[EE] is set) even 
during the secondary kemel initialization phase. It should be noted that even the 
secondaiy kemel initialization code is fully preemptable by the primary system. This is 
particularly important in order to do not disturb the primary operating system when a 
secondary operating system is restarted. 

Despite of enabled hardware interrapts, the virtualized exceptions 
(corresponding to hardware interrupts) are disabled when a secondary kemel is started. 
So, interrapts are not delivered by the nanokemel imtil they are expUcitly enabled by 
the secondary kemel at the end of the critical initialization phase. The software 
interrapts masking mechanism (based on virtual exceptions) is described in detail 
further in this document. 

The stack pointer is invalid when a secondary kemel is started. Usually, the 
secondary kemel uses a static initial stack located in the data section in order to 
execute its initialization code. 

Analogous to the primary, kemel, during the initialization phase, a secondary 
kemel typically invokes the nanokemel traps in order to attach handlers to the 
PowerPC and virtualized exceptions. Finally the secondary kemel enters in the idle 
loop and invokes the nanokemel idle trap. 

Intercepted Exceptions 

Li order to intercept a secondary exception, the nanokemel installs a pointer to 
its own handler into the corresponding entry of the hdlsO exception handler table in 
the secondary kemel context. Thus, when such an exception occurs, the exception 
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vector branches to the nanokemel handler which saves the secondary memory context 
and process the excq)tion. 

All intercepted exceptions can be classified according to its nature as interrupts, 
traps and programming exceptions (faults). 

The nanokemel intercepts all PowerPC interrapts (asynchronous exceptions) in 
order to forward them to the primary kernel: 

jfiT Reset 

^ Machine check 
^ System management 
^ External interrupt 
^ Performance monitor 
js^ Decrementer 
.es- Thermal 

Additionally, two nanokemel traps are performance critical and are handled 
specifically for secondary kernels. 

The first one sends a cross interrupt to the primary kemel. llie nanokemel 
processing is equivalent to the interrupt one except that the exception forwarded to the 
primary kemel corresponds to a sofiware intermpt rather than to a hardware one. So, 
like an intermpt, this trap preempts the current secondary kemel. 

The second one is called by a secondary kemel in order to process pending 
virtual exceptions, when they are enabled again. These traps take a part in the software 
intermpts masking mechanism and they are described in detail in the next section 
dedicated to the virtual exceptions. 

Analogous to the primary kemel, the nanokemel usually does not intercq>t 
programming exceptions except some special cases described below: 

js£ the floating point xmavailable exception (FPU) 

^ the vector imit unavailable exception (AltiVec VU) 

These exceptions are used by the nanokemel to implement a lazy mechanism 
of co-processing units sharing as described fiirther in this document 

Another special case is a debug agent which could be embedded in flie 
nanokemel in order to provide a host based remote systCTi debugging of the secondary 
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operating system. In this case, the debug agent usually intercepts some synchronous 
exceptions related either to debug features (e.g., single instruction trace) or to program 
errors (e.g., page fault). Such a debug agent design however is out of scope of this 
document. 

Virtaal Exceptions 

Virtual exceptions (VEX) is a mechanism provided by the nanokCTiel which 
allows a kernel to post an exception to a secondary kernel and to deliver it in a deferred 
maimer. In particular, the VEX mechanism is used in the PowerPC nanokemel 
architecture in order to replace hardware interrupts with software ones for a secondary 
kernel. 

The VEX interface consists in two field located in the kemel context: pending 
and enabled. These fields are meaningful only for a secondary kemel context but they 
are accessed by both the primary and secondary kernels. All virtual exceptions are 
naturally enumerated by the bit position in the pending (or enabled) field. So, there are 
in total 32 virtual exceptions supported by the nanokemel on the PowerPC architecture 
(the pending and enabled fields are 32 bit integer values). 



The table below shows how the virtual exceptions are mapped to the real ones: 



Virtual Exception 


Reat Exception 


Description 


0 


0x2 


Machine check 


1 


0x1 


Reset 


2 


0x14 


SMI 


3 


0x5 


External inteirtqpt 


4 


0x2e 


Performance monitor 


5 


0x9 


Decremeater 


6 


0x17 


Thennal 


7 


0x23 


XIRQ 


8 


0x21 


Debugger 


31 




Running 
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Virtual exceptions from 0 up to 6 are mq>ped to the PowerPC interrupts. The 
virtual exception 7 is mapped to the exception vector 0x23 used to deliver cross 
interrupts to tiie secondary kernel. The virtual exceptions from 9 up to 30 are not 
currently used and they are reserved for ftiture extensions. The virtual exception 31 
5 does not correspond to any real exception and it is in fact a pseudo virtual excq>tion 
which is used internally by the nanokemel is order to detect if the kemel is idle. How 
such a pseudo virtual exception works is described in detail ftirther in this document. 

Because multiple virtual exceptions can be pending at the same time but only 
one of them can be processed at a time, all virtual exceptions are prioritized according 

10 to its number. The highest priority is assigned to the Machine-check and the lowest 
priority is assigned to the Running pseudo exception. 

The pending VEX field of a secondary context is typically updated by the 
primary kemel which provides a driver for the virtual PIC device. Such a driver 
usually posts virtual exceptions (interrupts) to secondary kernels by setting appropriate 

1 5 bits in the pending VEX field. 

The enabled kEX" field is updated by the secondary kemel in order to enable or 
disable virtual exceptions. A given virtual exception is enabled if the corresponding bit 
is set in the enabled VEX field. Using the enabled VEX field, a secondary kemel 
implements critical sections protected against intemipts. In other words, a secondary 

20 kemel no more uses the MSR[EE] and MSR[ME] bits to disable/enable processor 
interrapts but rather modifies the enabled VEX&eld of its kemel context. 

A given virtual exception is delivered by the nanokemel if it is pending and 
enabled simultaneously. The nanokemel resets the corresponding pending bit just 
before jumping to the secondary exception handler. 

25 Note that, when porting a secondary kemel on the PowerPC nanokemel 

architecture, low-level exception handlers have to be modified in order to take into 
account the software intemipts masking mechanism which substitutes the hardware 
one. The hardware interrapts are always enabled at processor level when running a 
secondary kemel, except in low-level exception handler code, where processor state is 

30 saved using shared resources (scratch registers) requiring a critical section. In such a 
way, in the nanokemel environment, a secondary operating system becomes highly 
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preemptable by the primary operating system; it is pre-emptable except for the short 
periods when such low-level exception code is saving state. 

A virtual exception can be posted by the primary kernel while it is in disabled 
state. It this case, the exception is not delivered to the secondary kernel but it is rather 
kept pending until the exception is re-enabled again. So, when virtual exceptions are 
re-enabled by a secondary kernel, a check should be made wheflier any virtual 
exceptions are pending. If the check is positive, the secondary kernel should invoke the 
nanokamel in order to process such pending virtual exceptions. Such check is made by 
mean of the "process pending virtual exception" trap (nanokemel specific vector 
0x2500). 

In general, a secondary kernel re-enables virtual exceptions in two following 

cases: 

^ when virtual exceptions has been previously disabled by the secondary 
kernel in order to protect a critical section of code; 

js$ when virtual exceptions has been disabled while processing a processor 
exception low-level handler. 

Nanokernel Re-Entrance 

The nanokemel code is mostly executed with intemipts disabled at processor 
level preventing re-entrance into the nanokemel. On the other hand, some nanokemel 
invocations may take a long time and therefore the nanokemel has to enable uitermpts 
when executing such long operations in order to keep the primary interrapt latency 
low. 

There are two kinds of long nanokemel operations: 
js$ synchronous console output 

The operation duration depends on the serial line speed. For example, on a 
9600 baud rate line, a single character output may take up to 1 millisecond. 
js$ secondary kernel restart 

The operation duration depends on the kernel image size which is restored fix>m 

a copy. 



wo 2005/033928 



PCT/IB2004/003344 



47 

For all operations listed above, the nanokemel enables interrupts and therefore 

re-entrance from the primary kernel. On the other hand, while interrupts are enabled, 

« 

the nanokemel scheduler is never called in order to prevent another secondary kernel 
to be scheduled when returning from the primary inteiTiq)t handler. In other words, the 

5 nanokemel can be preempted by the primary kernel only (as result of an interrupt) but 
re-entrance from a secondary kernel is prohibited. Such a restriction allows the 
nanokemel to use global resources for the secondary execution environment. Thus, 
when the nanokemel is entered from a secondary operating system, it is not necessary 
to save all states, since it cannot be intermpted from a secondary operating system. 

10 The discussion above shows that the nanokemel must be capable to enable 

processor interrupts while executing code in the nanokemel memory context resulting 
from a trap called by the primary or a secondary kernel. In other words, the nanokemel 
must support a switch to the primary interrupt handler while running its own trap call 
handler. 

IS In order to support such a context switch nesting, the nanokemel manages a 

kemel context for itself; The nanokemel kemel context has a zero identifier value {ID 
field). It is used to save the nanokemel execution context (in hidden part) when 
switching to the primary kemel on interrapt. It is also used to store nanokemel specific 
intermpt handler pointers (in hdlsQ table). In this way, intemipts can be vectored 

20 tturougih the "currently" executing kmiel context even when the nanokemel code is 
ruiming. Additionally, the nanokemel supervisor stack is used to push critical shared 
processor ressources as well as a part of the nanokemel and primary kemel contexts 
that would otherwise be overwritten on primary kemel re-entrance. 

The nanokemel uses an interrupt prolog/epilog pair of routines to handle an 

25 interrapt in the nanokemel and to retum to nanokemel interrapted code at the end of 
the interrupt processing by the primary kernel. Note that the prolog/epilog pair of 
routines is different depending on the last kemel that has been calling a nanokemel 
trap. Indeed the kind and amount of information to be pushed on the stack is not the 
same for the primary or a secondary trap calla:. 
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The interrupt prolog routine is the nanokemel interrupt handler. It is attached to 
all interrupt entries of the hdlsjj exception handler table in the nanokemel kernel 
context. This handler performs the following actions: 

jss save part of the kernel contexts into an interrupt frame on the stack, 
5 jg^ update kemel contexts wilh current values to handle primary kernel re- 

entrance, 

js$ modify srrO register value to retum to the appropriate epilog routine, 
js$ switch to the corresponding primary kemel interrapt handler. 
The interrupt epilog routine is called when returning from the primary kemel 
10 interrupt processing (rfi). It is used to directly restore the nanokemel interrupted 
execution context without calling the nanokemel schedvder, and goes througji the 
following steps: 

jss restore current state from kemel contexts, 

restore kemel contexts from the intermpt stack frame (saved hy the prolog 

15 routine), 

js$ retum to the interropted code. 

The nanokemel intermpt stack frame is used to save the following information: 
js$ copy of critical part of the nanokemel kemel context: hr, rl, r2, srrO, srrl 
registers, 

20 the last trap caller (pointer to a primary or secondary kemel context). 

jss only if last trap caller is the primary: copy of critical part of the primary 
kemel context: rl, r2, srrO, srrl registers. 

The nanokemel defines a pair of functions which are used to enable/disable 
intermpts. These functions respectively save and restore in the stack an additional part 
25 of the nanokemel execution context which is not systematically saved at trap entry. 

When enabling intermpt, the nanokemel goes through the following steps: 

je$ save scratch registers in the stack (sprgO-3), 

js$ save current pair of intermpt prolog/epilog routines in the stack, 

jes update current pair of interrupt prolog/epilog routines according to trap 
30 caller primary or secondary), 

jes enable intermpts at processor level (set MSR[EE] bit). 
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When disabling interrupt, the nanokemel goes through the following stq)s: 
^ disable interrupts at processor level (clear MSR[EE] bit), 

restore current pair of intOTupt prolog/epilog routines from the stack, 
js$ restore scratch registCTs from the stack (sprgO-3). 
S The figure 4 shows the execution flow and how the nanokemel stack is used to 

allow intermpt handling and primary kernel re-entrance. 



Scheduler 

The main role of an operating system scheduler is to choose the next task to 

10 run. Because the nanokemel controls execution of operating systems, the nanokemel 
scheduler chooses the next secondary operating system to run. hi other words, the 
nanokemel adds an extra scheduling level to the whole system. 

Note that, in the nanokemel architecture, the primary operating system has a 
higher priority level with respect to secondary systems and the CPU is given to a 

15 secondary system only when the primary one is in the idle loop. We can say that the 
primary kernel is not preemptable and it explicitly invokes the nanokemel schedider 
through the idle method called in the idle loop. Once an interrapt occurs when running 
a secondary system, the primary kernel intermpt handler is invoked. From the primary 
kernel perspective, such an intermpt preempts the background thread executing the 

20 idle loop. Once the interrupt is handled and all related tasks are done, the primary 
kernel retxmis to tiie nanokemel which invokes the nanokemel scheduler in order to 
determine the next secondary system to run. From the primary perspective, the kernel 
just returns to the background thread preempted by the interrupt. The secondary 
activity is transparent for the primary kemel and it does not change the primary system 

25 behavior. 

The nanokemel may implement different scheduling policies. By default, 
however, a priority based algoritiun is used. Note that, at the same priority level, the 
nanokemel uses a rotmd-robin scheduling policy. Priority of a given secondary kemel 
is statically configured at system image build time. 
30 Whatever the in:q)lemented scheduling policy is, the scheduler has to detect 

whether a given secondary system is ready to run. This condition is calculated as the 
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bitwise logical and operation between the pending VEXdiid enabled FEAT fields of the 
kernel context. A non zero result indicates that the system is ready to run. 

As was described above, each bit in the pending VEX and enabled FE^^pair 
represents a virtual exception. Rephrasing the ready to run criteria, we can say that a 
5 secondary system is m the ready to run state if there is at least one non masked pending 
virtual exception. 

Among all virtual exceptions which are typically mapped to the hardware and 

software (cross) interrupts, there is a special virtual exception (running) reflecting 

whether the kemel is currently idle. 
10 The running bit is cleared in the pending VEX field each time a secondary 

kemel invokes the idle method and the running bit is set in the pending FEAT field each 

time a virtual exception is deUvered to the secondary kemel. 

The running bit is normally always set in the enabled J^EAT field for a running 

secondary kemel. The nanokemel sets this bit when a secondary kemel is started and it 
15 resets this bit when a secondary kemel is halted. The secondary kemel should never 

clear the running bit wh^ masking/unmasking intermpts mapped to virtual 

exceptions. 

Note that an external agent is able to suspend/resume execution of a secondary 
kemel by clearing/restoring the enabled VEX field in its kemel context This feature 

20 opens possibilities for a scheduling poUcy agent to be implemented outside of the 
nanokemel, as a primary kemel task. In addition, this also enables a debug agent for a 
secondary kemel to be running as a task on top of the primary kemel. An advantage of 
such a secondary debug agent is that all services provided by the primary operating 
system become available for debugging (e.g., networking stack) and the secondary 

25 kemel debugging may be done concurrently with critical tasks running on the primary 
operating system. 



Cross Interrupts 

This section mostly consolidates information (already given in previous 
30 sections) related to the nanokemel cross interrupts mechanism. 

Two following kinds of cross intermpts will be considered here: 
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jg^ a cross int^rupt sent to a secondary kernel 
^ a cross interrupt sent to the primary kernel 

In ord^ to send a cross interrupt to a destination secondary kernel, a source 
kernel first sets a bit corresponding to the cross interrupt source in tihe pending XIRQ 
field of tiie destination 

kernel context. Then the source kernel posts the cross interrupt VEX to the 
destination kernel setting the corresponding bit in the pending VEX field of the 
destination kernel context. Once the cross interrupt handler is called by the nanokemel, 
it checks the pending XIRQ field, clears bit corresponding to the pending cross 
interrupt source and finally invokes handlers attached to this source. Both source and 
destination kernels uses atomic instructions to update the pending XIRQ field. Note 
that the same algorithm is used by both types of source kernel: primary and secondary. 

In order to send a cross interrupt to the primary kemel, a secondary kernel first 
sets a bit corresponding to tiie cross intermpt source in the pending XIRQ field of the 
primary kemel context. Then the secondary kernel invokes the nanokemel executing 
the XIRQ trap. The nanokemel immediately preempts the secondary kemel and 
invokes the primary low-level cross interrapt handler which checks the pending XIRQ 
field, clears bit corresponding to the pending cross intermpt source and finally invokes 
handlers attached to this source. 

The cross interrapt zero must not be used by kernels. This interrapt is reserved 
for the nanokemel to notify kernels that a halted kemel has been started or a running 
kemel has been halted. In other words, the cross interrapt zero notifies running kernels 
that the global system configuration is changed. It is broad casted to all running kernels 
each time the state of the running bit field pointed to by the kemel contexts, is 
changed. 

Co-processing Units Management 

The co-processing units (FPU, SIMD xmit) are computing resources which are 
typically shared by all operating systCTois running in the nanokemel environment. In the 
following the floating-point unit (FPU) is taken as an example. However other co- 
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processing units like AltiVec vectorial computing unit (SIMD) are managed in same 
way. 

On PowerPC architecture, the nanokemel manages co-processing units sharing 
in a lazy mamier. This means, for example, that when a switch from one operating 
5 system to another occurs, the FPU is not immediately given to the newly scheduled 
operating system, instead, the FPU context switch is deferred imtil the newly 
scheduled system really executes floating-point instructions and accesses floating- 
point registers. 

Such a lazy FPU dispatching algorithm allows the nanokemel to reduce the 

10 system switch time. This is especially important in order to reduce the primary 
interrupt latency because FPU is normally not used at interrupt level and therefore it is 
usually not necessary to save and restore FPU registers in order to preempt a secondary 
operating system and to call a primary interrupt handler. 

The nanokemel handles an FPU owner global variable pointiag to the context 

15 of the kernel which currently uses FPU. In case there is no FPU owner, the FPU owner 
context is set to zero. An FPU context is located in the hidden part of the kernel 
context. Such a context keeps state of tiie FPU engine (i.e., floating-point registers and 
status) when the kemel is not FPU owner. Obviously, the state of the FPU owner is 
kept by the FPU engine hardware. When the nanokemel changes the FPU owner, the 

20 FPU state is saved to the old FPU context and restored from the new one. 

The nanokemel uses the floating-point unavailable (FP) bit of the MSR register 
in order to provoke an exception when FPU is used by a non FPU owner. The MSR 
register image takes a part in the hidden part of the kemel context. The MSR[FP] bit is 
cleared in the previous owner context, and set in the newly owner context, at FPU 

25 switch time. The FP bit is cleared in all kemel contexts excqpt the FPU owner where it 
is set. This allows for the nanokemel to iutercept the floating-point imavailable 
exception (0x8) for all non FPU owners while the FPU owner handles this exception in 
a native way. 

An FPU switch occurs when the nanokemel intercepts the 'TP unavailable" 
30 exception (this requires a modification to tiie operating system kemel). La order to 
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switch the FPU engine between two kernels, the nanokemel releases the current FPU 
owner and assigns the new one. 

In order to release the current FPU owner, the nanokemel saves the current 
FPU state in its kernel context together with the current state of the MSR[FP] bit and 
clear the FP bit in the MSR register image. In addition, the nanokemel installs its own 
exception handler into the appropriate entry of tiie hdlsO exception handler table in the 
kernel context, in order to intercept the FP unavailable exception (0x8). The native 
exception handler pointer is saved in a savedHdlsO table into the hidden part of the 
kemel context. 

In order to assign a new FPU owner, the nanokemel restores the FPU state 
from the kemel context and the previously saved state of the FP bit into the MSR 
image. In addition, the native exception handler is restored from the savedHdlsO table 
into the hdlsO table of the kemel context, in order to handle the FP unavailable 
exception in a native way while owning FPU. 

Because the nanokemel uses the FP bit of the MSR register in order to 
implement a lazy FPU swicch, a kemel is not allowed to change the state of this bit 
while it is not owner of the FPU (outside of the native FP unavailable exception 
handler). In particular, this means fliat the FPU emulation is not supported by a kemel 
ported to the nanokemel architecture. 

Note that usually an operating system kemel also uses the FP bit of the MSR 
register in order to implement a lazy FPU switch between processes. Because the MSR 
registCT image takes a part in the kemel context and therefore it is saved and restored at 
system switch, tiie native FPU management can be kept almost unchanged in the 
nanokemel environment. 

The same mechanisms are used for the vectorial computing imits integrated 
into some PowerPC processor implementations (AltiVec). In that case the VU bit of 
the MSR register plays the same role as the FP bit does for the FPU. Also, the vector 
unavailable exception (vector OxOfZO) is intercepted to implement context switch of 
the SIMD unit in a lazy marmer. 
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Other aspects and embodiments 

It will be clear from the forgoing that the above-described embodiments are 
only examples, and that many other embodiments are possible. The operating systems, 
platforms and programming techniques mentioned may all be freely varied. Any other 
5 modifications, substitutions and variants which would be apparent to the skilled person 
are to be considered within the scope of the invention, whether or not covered hy the 
claims which follow. For the avoidance of doubt, protection is sought for any and all 
novel subject matter and combinations thereof disclosed herein. 
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